⬡ Hub
Skip to content

AWS Glacier (for Archival Storage)

Detailed Content

Amazon S3 Glacier is a secure, durable, and extremely low-cost cloud storage service for data archiving and long-term backup. It is designed for data that is infrequently accessed and where retrieval times of several minutes to several hours are acceptable.

Core Concepts

  • Vaults: The primary container for archives in Glacier. You create vaults in a specific AWS region, and each vault can contain an unlimited number of archives. Vaults provide a way to organize your archives and apply access policies and retrieval policies.
  • Archives: The fundamental unit of storage in Glacier. An archive can be any data, such as photos, videos, documents, or backup files. Each archive has a unique ID and an optional description. Archives are immutable once uploaded.
  • Retrieval Options: Glacier offers different retrieval options based on your urgency and cost requirements. These options apply to both S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive:
    • Expedited: Fastest retrieval (typically 1-5 minutes), highest cost. Suitable for urgent requests for a subset of your data.
    • Standard: Default retrieval (typically 3-5 hours for S3 Glacier Flexible Retrieval, 12 hours for S3 Glacier Deep Archive), moderate cost. Suitable for less urgent needs.
    • Bulk: Slowest retrieval (typically 5-12 hours for S3 Glacier Flexible Retrieval, 48 hours for S3 Glacier Deep Archive), lowest cost. Suitable for retrieving large amounts of data where time is not critical.
  • Data Retrieval Policy: You can set a data retrieval policy for each vault to manage your retrieval costs. Options include:
    • Free Tier Only: Prevents retrievals that exceed the free tier.
    • Max Retrieval Rate: Sets a maximum data retrieval rate in GB per hour.
    • No Retrieval Limit: Allows unlimited retrievals.
  • Vault Lock: Allows you to easily deploy and enforce compliance controls for individual S3 Glacier vaults with a WORM (Write Once Read Many) model. Once locked, the policy cannot be changed, ensuring data immutability for regulatory compliance (e.g., SEC Rule 17a-4, FINRA Rule 4511).
  • Inventory: A list of all archives in a vault. Glacier automatically generates a vault inventory once a day. You can initiate a job to retrieve this inventory, which typically takes 3-5 hours.
  • S3 Glacier Deep Archive: The lowest-cost storage class in AWS, designed for long-term archival of data that is accessed rarely (e.g., once or twice a year). It offers retrieval times of 12 hours (Standard) or 48 hours (Bulk).

Integration with S3

While Glacier can be used directly, it is most commonly used in conjunction with Amazon S3 through S3 Lifecycle policies. This allows you to store data in S3 for immediate access and then automatically transition it to Glacier (or S3 Glacier Deep Archive) for cost-effective long-term archiving as the data ages and becomes less frequently accessed. This approach provides a tiered storage solution, optimizing both accessibility and cost.

  • S3 Standard to S3 Glacier: Data is initially stored in S3 Standard for frequent access, then automatically moved to S3 Glacier after a defined period (e.g., 30, 60, 90 days).
  • S3 Standard-IA/One Zone-IA to S3 Glacier: Data can also transition from Infrequent Access storage classes to Glacier.
  • S3 Intelligent-Tiering: This S3 storage class automatically moves data between four access tiers (Frequent, Infrequent, Archive, Deep Archive) based on access patterns, including S3 Glacier and S3 Glacier Deep Archive, without performance impact or operational overhead.

Use Cases

  • Data Archiving and Long-Term Backup: The primary use case for Glacier is storing data that is infrequently accessed but must be retained for long periods. This includes backups of databases, application data, and system images.
  • Regulatory and Compliance Archives: Many industries (e.g., healthcare, finance) have strict regulations requiring data to be retained for many years. Glacier provides a secure and cost-effective solution for this, especially with Vault Lock to enforce WORM policies.
  • Media Asset Archiving: Storing large volumes of media content, such as raw video footage, high-resolution images, and broadcast archives, where immediate access is not required.
  • Scientific Data Archiving: Archiving large datasets from scientific research, experiments, and simulations that need to be preserved for future analysis or validation.
  • Digital Preservation: Used by libraries, museums, and government agencies for the long-term preservation of digital records, documents, and cultural heritage assets.
  • Magnetic Tape Replacement: Serves as a modern, cloud-based alternative to on-premises magnetic tape libraries, reducing physical infrastructure management and offering higher durability.

Interview Questions

Conceptual Questions

  1. What is AWS Glacier and what is its primary purpose? How does it differ from S3 Standard?
    • AWS Glacier is a secure, durable, and extremely low-cost cloud storage service primarily designed for data archiving and long-term backup. Its main purpose is to store infrequently accessed data for long periods, where retrieval times of several minutes to several hours are acceptable.
    • Difference from S3 Standard: S3 Standard is for frequently accessed, general-purpose object storage with immediate retrieval. Glacier is for archival, with much lower storage costs but higher retrieval costs and longer retrieval times.
  2. Explain the different data retrieval options in Glacier (Expedited, Standard, Bulk) and when you would use each.
    • Expedited (1-5 minutes): Fastest retrieval, highest cost. Use for urgent, small retrievals where you need immediate access to a subset of your archived data (e.g., critical legal documents, a single backup file for a quick restore).
    • Standard (3-5 hours for S3 Glacier Flexible Retrieval, 12 hours for S3 Glacier Deep Archive): Default retrieval, moderate cost. Suitable for less urgent needs where you can wait a few hours for your data (e.g., daily backups, historical data analysis).
    • Bulk (5-12 hours for S3 Glacier Flexible Retrieval, 48 hours for S3 Glacier Deep Archive): Slowest retrieval, lowest cost. Use for retrieving large amounts of non-urgent data where time is not critical (e.g., restoring an entire archive, large-scale data migrations).
  3. What is a Glacier Vault Lock and why is it important for compliance?
    • Glacier Vault Lock allows you to easily deploy and enforce compliance controls for individual S3 Glacier vaults with a WORM (Write Once Read Many) model. Once a Vault Lock policy is applied and locked, it becomes immutable, meaning data cannot be changed or deleted for a specified period. This is crucial for meeting regulatory compliance requirements (e.g., SEC Rule 17a-4, FINRA Rule 4511) that mandate data retention and immutability.
  4. How does S3 Glacier integrate with Amazon S3? What are S3 Lifecycle Policies in this context?
    • S3 Glacier integrates seamlessly with Amazon S3 primarily through S3 Lifecycle Policies. This allows you to store data in S3 for immediate access and then automatically transition it to S3 Glacier (or S3 Glacier Deep Archive) for cost-effective long-term archiving as the data ages and becomes less frequently accessed. Lifecycle policies define rules to transition objects to different storage classes or expire them after a certain period.
  5. What is S3 Glacier Deep Archive and when would you choose it over S3 Glacier Flexible Retrieval?
    • S3 Glacier Deep Archive is the lowest-cost storage class in AWS, designed for long-term archival of data that is accessed rarely (e.g., once or twice a year). You would choose it over S3 Glacier Flexible Retrieval when:
      • Cost is the absolute highest priority for archival storage.
      • You can tolerate longer retrieval times (Standard: 12 hours, Bulk: 48 hours).
      • The data is truly cold and very infrequently accessed.
  6. How do you get a list of all archives stored in a Glacier vault?
    • You initiate a vault inventory retrieval job. Glacier automatically generates a vault inventory once a day. You can request this inventory, and it typically takes 3-5 hours to complete. Once the job is complete, you can download the inventory list, which contains details about all archives in the vault.

Scenario-Based Questions

  1. Your company needs to archive financial records for 10 years to meet strict regulatory requirements. These records are rarely accessed, but when they are, a retrieval time of up to 12 hours is acceptable. Cost optimization is paramount. What AWS service and configuration would you use?
    • I would use Amazon S3 Glacier Deep Archive. This service offers the lowest storage cost for long-term archival. For retrieval, I would use the Standard retrieval option (12 hours) as it meets the acceptable retrieval time and is more cost-effective than Expedited. To enforce the 10-year retention, I would implement a Glacier Vault Lock policy on the vault, setting a WORM (Write Once Read Many) compliance mode for 10 years.
  2. You have a large dataset (petabytes) of historical sensor data stored in Amazon S3 Standard. This data is accessed frequently for the first 30 days, then infrequently for the next 60 days, and finally needs to be archived for 5 years. How would you manage this data lifecycle efficiently and cost-effectively?
    • I would use S3 Lifecycle Policies on the S3 bucket:
      • Rule 1: Transition objects from S3 Standard to S3 Standard-IA after 30 days.
      • Rule 2: Transition objects from S3 Standard-IA to S3 Glacier Flexible Retrieval after 90 days (30 + 60 days).
      • Rule 3: Transition objects from S3 Glacier Flexible Retrieval to S3 Glacier Deep Archive after 1 year (or a suitable period based on access patterns within Glacier Flexible Retrieval).
      • Rule 4: Expire objects after 5 years in S3 Glacier Deep Archive. This tiered approach optimizes costs by moving data to progressively cheaper storage classes as its access frequency decreases.
  3. You have a critical backup of your production database that needs to be stored in Glacier. In case of a disaster, you need to restore this backup as quickly as possible, ideally within minutes. Which retrieval option would you configure for this specific archive and what are the cost implications?
    • For this critical backup, I would use the Expedited retrieval option. This provides the fastest retrieval time (1-5 minutes), which is crucial for minimizing downtime during a disaster recovery scenario. The cost implication is that Expedited retrievals are the most expensive per GB compared to Standard or Bulk retrievals, but the priority in this scenario is speed of recovery over cost.
  4. Your team needs to perform an audit of all data archived in a specific Glacier vault. They need a list of all archives, their sizes, and creation dates. How would you provide this information?
    • I would initiate a vault inventory retrieval job for the specific Glacier vault. This job would generate a manifest of all archives within the vault. Once the job completes (typically 3-5 hours), I would download the output, which is a JSON file containing the archive IDs, descriptions, creation dates, and sizes. This information can then be used for the audit.

Coding/CLI Examples

Here are some common Glacier operations using the AWS CLI and Python (Boto3).

AWS CLI Examples

  1. Create a Glacier vault: bash aws glacier create-vault \ --vault-name my-archive-vault-cli \ --account-id - # Use '-' for the current account

  2. Upload an archive to a Glacier vault: ```bash # Create a dummy file for upload echo "This is my important document content." > my-document.txt

    aws glacier upload-archive \ --vault-name my-archive-vault-cli \ --body my-document.txt \ --archive-description "My important document backup from CLI" \ --account-id - \ --query 'archiveId' --output text ```

  3. Initiate a job to retrieve an archive from Glacier (Standard retrieval): ```bash VAULT_NAME="my-archive-vault-cli" ARCHIVE_ID="your-archive-id" # REPLACE with the actual Archive ID from upload

    aws glacier initiate-job \ --vault-name $VAULT_NAME \ --account-id - \ --job-parameters '{ "Type": "archive-retrieval", "ArchiveId": "'"$ARCHIVE_ID"'", "Description": "Retrieve my important document", "Tier": "Standard" }' ```

  4. Download the output of a completed retrieval job: ```bash VAULT_NAME="my-archive-vault-cli" JOB_ID="your-job-id" # REPLACE with the Job ID returned from initiate-job

    First, check job status (optional, but good practice)

    aws glacier describe-job \ --vault-name $VAULT_NAME \ --account-id - \ --job-id $JOB_ID

    If job status is 'Succeeded', download output

    aws glacier get-job-output \ --vault-name $VAULT_NAME \ --account-id - \ --job-id $JOB_ID \ output.txt ```

  5. Initiate a vault inventory retrieval job: ```bash VAULT_NAME="my-archive-vault-cli"

    aws glacier initiate-job \ --vault-name $VAULT_NAME \ --account-id - \ --job-parameters '{ "Type": "inventory-retrieval", "Description": "Get vault inventory" }' ```

Python (Boto3) Examples

First, ensure you have Boto3 installed (pip install boto3) and your AWS credentials configured.

  1. Create a Glacier vault: ```python import boto3

    glacier_client = boto3.client('glacier')

    vault_name = "MyBoto3ArchiveVault"

    try: response = glacier_client.create_vault(vaultName=vault_name) print(f"Created Glacier vault: {vault_name}") except Exception as e: print(f"Error creating vault: {e}") ```

  2. Upload an archive to a Glacier vault: ```python import boto3 import os

    glacier_client = boto3.client('glacier')

    vault_name = "MyBoto3ArchiveVault" file_path = "my_boto3_document.txt" archive_description = "Important document from Boto3"

    Create a dummy file

    with open(file_path, "w") as f: f.write("This is the content of my important document uploaded via Boto3.")

    try: with open(file_path, 'rb') as f: response = glacier_client.upload_archive( vaultName=vault_name, archiveDescription=archive_description, body=f.read() ) archive_id = response['archiveId'] print(f"Uploaded archive with ID: {archive_id}") except Exception as e: print(f"Error uploading archive: {e}") finally: os.remove(file_path) ```

  3. Initiate an archive retrieval job (Expedited): ```python import boto3

    glacier_client = boto3.client('glacier')

    vault_name = "MyBoto3ArchiveVault" archive_id = "your-archive-id" # REPLACE with the actual Archive ID

    try: response = glacier_client.initiate_job( vaultName=vault_name, jobParameters={ 'Type': 'archive-retrieval', 'ArchiveId': archive_id, 'Description': 'Expedited retrieval for critical data', 'Tier': 'Expedited' } ) job_id = response['jobId'] print(f"Initiated expedited retrieval job with ID: {job_id}") except Exception as e: print(f"Error initiating retrieval job: {e}") ```

  4. Initiate a vault inventory retrieval job: ```python import boto3

    glacier_client = boto3.client('glacier')

    vault_name = "MyBoto3ArchiveVault"

    try: response = glacier_client.initiate_job( vaultName=vault_name, jobParameters={ 'Type': 'inventory-retrieval', 'Description': 'Retrieve vault inventory via Boto3' } ) job_id = response['jobId'] print(f"Initiated inventory retrieval job with ID: {job_id}") except Exception as e: print(f"Error initiating inventory retrieval job: {e}") ```